Papers with model calibration
Page Stream Segmentation with LLMs: Challenges and Applications in Insurance Document Automation (2025.coling-industry)
Copied to clipboard
| Challenge: | Existing models for page Stream Segmentation (PSS) are inadequate for high-stakes applications. |
| Approach: | They propose to use large language models for page Stream Segmentation to apply parameter-efficient fine-tuning to real-world insurance data. |
| Outcome: | The proposed model outperforms baseline models in page- and stream-level segmentation accuracy. |
A Close Look into the Calibration of Pre-trained Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) may fail in giving reliable estimates of their predictive uncertainty. |
| Approach: | They conduct fine-grained control experiments to study the dynamic change in PLMs’ calibration performance in training. |
| Outcome: | The proposed methods significantly reduce PLMs’ confidence in wrong predictions. |
CATfOOD: Counterfactual Augmented Training for Improving Out-of-Domain Performance and Calibration (2024.eacl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) have shown remarkable generalization capabilities, performing well on various tasks such as question answering (QA), complex reasoning, and code generation. |
| Approach: | They propose to augment training data of smaller language models with automatically generated counterfactuals (CF) instances to improve out-of-domain (OOD) performance of SLMs in extractive question answering setup. |
| Outcome: | The proposed approach improves out-of-domain (OOD) performance of small language models in extractive question answering setup. |
Extracting Victim Counts from Text (2023.eacl-main)
Copied to clipboard
| Challenge: | Using tagging and regex methods, data on injured, displaced, or abused victims is difficult . data on earthquake injuries and deaths is scarce, subjective, or biased . |
| Approach: | They compare tagging approaches to extract injured, displaced, or abused victims . they discuss calibration and investigate out-of-distribution and few-shot performance . |
| Outcome: | The proposed model is among the first to apply numeracy-focused large language models in a real-world use case with a positive impact. |
Calibrating Zero-shot Cross-lingual (Un-)structured Predictions (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing need for model calibration when natural language models are deployed in critical tasks. |
| Approach: | They compare model calibration methods in a context of zero-shot cross-lingual transfer with pre-trained language models. |
| Outcome: | The proposed method fails to calibrate more complex confidence estimations in structured predictions compared to expressive alternatives like Gaussian Process Calibration. |
Reasoning about Uncertainty: Do Reasoning Models Know When They Don’t Know? (2026.findings-eacl)
Copied to clipboard
| Challenge: | Reasoning models are prone to generating confident, plausible responses that are incorrect (hallucinations). |
| Approach: | They introduce introspective uncertainty quantification to examine whether reasoning models are well-calibrated and does deeper reasoning improve their calibration? |
| Outcome: | The proposed model calibrations show that models are overconfident, overconfent and overconfust with deeper reasoning. |
STAR: Constraint LoRA with Dynamic Active Learning for Data-Efficient Fine-Tuning of Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing studies show that supervised training is still necessary for complex reasoning tasks. |
| Approach: | They propose a method to integrate uncertainty-based active learning and LoRA to effectively integrate the two methods. |
| Outcome: | The proposed approach outperforms baseline models on three reasoning tasks. |
RoAST: Robustifying Language Models via Adversarial Perturbation with Selective Training (2023.findings-emnlp)
Copied to clipboard
Jaehyung Kim, Yuning Mao, Rui Hou, Hanchao Yu, Davis Liang, Pascale Fung, Qifan Wang, Fuli Feng, Lifu Huang, Madian Khabsa
| Challenge: | Several perspectives of robustness for pre-trained language models have been studied independently, but lacking a unified consideration in multiple perspectives. |
| Approach: | They propose a technique to enhance the multi-perspective robustness of LMs by introducing adversarial perturbation while the model parameters are selectively updated upon their relative importance. |
| Outcome: | The proposed technique improves the robustness of LMs by incorporating four perspectives on model robustness. |
Test-time Augmentation for Factual Probing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to improve factual probing are relation-specific and do not generalize to unseen relation types. |
| Approach: | They propose to use test-time augmentation to augment and ensemble prompts at test time to reduce sensitivity to prompt variations. |
| Outcome: | The proposed method improves model confidence, but for other models, it leads to degradation. |
On the Calibration of Pre-trained Language Models using Mixup Guided by Area Under the Margin and Saliency (2022.acl-long)
Copied to clipboard
| Challenge: | Existing studies have shown that mixing up can improve model calibration on image classification tasks, but little is known about using it on natural language understanding (NLU) tasks. |
| Approach: | They propose a mixup strategy for pre-trained language models that improves model calibration further by using the AUM statistic and saliency map. |
| Outcome: | The proposed mixup improves model calibration on natural language understanding tasks while maintaining competitive accuracy. |
On the Risk of Evidence Pollution for Malicious Social Text Detection in the Era of LLMs (2025.acl-long)
Copied to clipboard
| Challenge: | Evidence-enhanced detectors are able to detect malicious social text, but they are prone to evidence pollution. |
| Approach: | They propose three defense strategies to mitigate evidence pollution by large language models by machine-generated text detection and a mixture of experts. |
| Outcome: | The proposed defense strategies could mitigate evidence pollution, but they faced limitations for practical employment. |
Boundary Smoothing for Named Entity Recognition (2022.acl-long)
Copied to clipboard
| Challenge: | Named entity recognition models often encounter over-confidence issues . boundary smoothing is a method that re-assigns entity probabilities from annotated spans to the surrounding ones . |
| Approach: | They propose a method for regularizing entity probabilities from annotated spans to the surrounding ones. |
| Outcome: | The proposed method achieves better than or competitive with previous state-of-the-art systems on well-known benchmarks. |
Beyond Noise: Mitigating the Impact of Fine-grained Semantic Divergences on Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Prior work treats all types of mismatches between source and target as noise . Consequently, it remains unclear how noisy parallel training samples impact NMT training. |
| Approach: | They propose a divergent-aware NMT framework that uses factors to help NMT recover from the degradation caused by naturally occurring divergences. |
| Outcome: | The proposed framework improves translation quality and model calibration on EN-FR tasks. |
Local Temperature Beam Search: Avoid Neural Text DeGeneration via Enhanced Calibration (2023.findings-acl)
Copied to clipboard
Dongkyu Lee, Gyeonghun Kim, Janghoon Han, Taesuk Hong, Yi-Reun Kim, Stanley Jungkyu Choi, Nevin L. Zhang
| Challenge: | Existing approaches to inference have been based on stochastic decoding but they sacrifice output quality due to randomness. |
| Approach: | They propose a deterministic decoding scheme, local temperature beam search, which reduces repetition while maintaining the level of coherence as in beam search. |
| Outcome: | The proposed inference scheme reduces repetition while maintaining coherence as in beam search. |
On the Calibration of Large Language Models and Alignment (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models are becoming more popular and are proving to be reliable . however, their reliability is often understudied due to their uncertainty and complex structure . |
| Approach: | They conduct a systematic examination of the calibration of aligned language models throughout the entire construction process including pretraining and alignment training. |
| Outcome: | The results shed light on whether popular large language models are well-calibrated and how the training process influences model calibration. |
Hard Gate Knowledge Distillation - Leverage Calibration for Robust and Reliable Language Model (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing knowledge distillation schemes focus on a teacher as a source of knowledge and a gauge to detect miscalibration of a student. |
| Approach: | They propose a method that uses a teacher model as a source of knowledge and a model as an error detector to detect miscalibration of a student. |
| Outcome: | The proposed scheme improves model generalization and significantly lowers calibration error. |
GRACE: A Granular Benchmark for Evaluating Model Calibration against Human Calibration (2025.acl-long)
Copied to clipboard
| Challenge: | Language models are often miscalibrated, leading to confidently incorrect answers. |
| Approach: | They propose a benchmark for language model calibration that incorporates comparison with human calibration. |
| Outcome: | The proposed metric analyzes model calibration errors and identifies types of miscalibration that differ from human behavior. |
Mind the Gap: A Closer Look at Tokenization for Multiple-Choice Question Answering with LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have highlighted the significant performance variation that can arise from minor changes in prompt design. |
| Approach: | They propose to tokenize the space following the colon to facilitate automated answer extraction via next-token probabilities. |
| Outcome: | The proposed tokenization improves model calibration and improves confidence estimates. |
COM-BOM: Bayesian Exemplar Search for Efficiently Exploring the Accuracy-Calibration Pareto Frontier (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior exemplar selection methods focus on maximizing predictive accuracy, neglecting model calibration. |
| Approach: | They propose to use a Bayesian optimization algorithm to optimize for predictive accuracy and calibration. |
| Outcome: | The proposed algorithm beats or matches baselines on multiple tasks from un-saturated MMLU-pro benchmarks while requiring minimal number of API calls. |
Model Calibration for Emotion Detection (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a MixUp method is used to calibrate emotion detection models based on knowledge distillation and the MixUp data augmentation technique. |
| Approach: | They propose a method that uses knowledge distillation and the MixUp data augmentation technique to enhance the trustworthiness of emotion detection models. |
| Outcome: | The proposed method improves the accuracy of the teacher models and the student models. |